Weakly supervised and explainable training of a machine-learning-based named-entity recognition (NER) mechanism

ABSTRACT

Systems and methods for weakly-supervised training a machine-learning model to perform named-entity recognition. All possible entity candidates and all possible rule candidates are automatically identified in an input data set of unlabeled text. An initial training of the machine-learning model is performed using labels assigned to entity candidates by a set of seeding rules as a first set of training data. The trained machine-learning model is then applied to the unlabeled text and a subset of rules from the rule candidates is identified that produces labels that most accurately match the labels assigned by the trained machine-learning model. The machine-learning model is then retrained using the labels assigned by the identified subset of rules as the second set of training data. This process is iteratively repeated to further refine and improve the performance of the machine-learning model for named-entity recognition.

BACKGROUND

The present invention relates to systems and methods for performingnamed-entity recognition (NER) using machine-learning techniques and,more specifically, for training named-entity recognition (NER) models.

SUMMARY

Named-entity recognition (NER) is a mechanism in which automatedprocessing (e.g., computer-based processing) is applied to unstructuredtext in order to identify and categorize occurrences of “named entities”(e.g., people, businesses, locations, etc.) in the unstructured text.For example, in some implementations, NER is a machine-learning-basednatural language processing mechanism in which unstructurednatural-language sentences are provided as input to a machine-learningmodel and the output of the machine-learning model includes anindication of an assigned category for each “entity” (or potentialentity) in the sentence (e.g., words or phrases that appear in thesentence that the machine-learning model determines may correspond toproper names, objects, etc.). For example, if the input sentenceprovided to as input recites: “John is travelling to London,” the outputof a trained NER machine-learning model may indicate the “John” iscategorized as a “person” and “London” is categorized as a “location.”

In some implementations, named-entity recognition (NER) is an essentialtask for many downstream information extraction tasks (e.g., relationextraction) and knowledge base construction. Supervised training ofnamed-entity recognition has achieved reliable performance due, forexample, to advances in deep neural models. However, supervised trainingof an NER model requires a large amount of manual annotation of data fortraining. This can require significant amounts of time in all cases butis particularly challenging in some specific domains and/or whentraining an NER model for low resource languages, where domain-expertannotation is difficult to obtain.

In some implementations, “distantly supervised” training is used toautomatically generate labeled data from open knowledge bases ordictionaries. Distant supervision makes it possible to generate trainingdata for NER models at a large scale without expensive human efforts.However, all distantly supervised methods rely on an existing knowledgebase or dictionary and, in some cases, an open knowledge base is notavailable (e.g., in the biomedical field, technical documents, etc.).

Accordingly, in some implementations, the systems and methods describedherein provide a “weakly supervised” mechanism for training amachine-learning NER model. In the weakly-supervised approach, a smallset of symbolic rules—referred to herein as “seeding rules”—is used tolabel data in unstructured text. In some implementations, the seedingrules and their associated labels may be provided or defined manuallyfor a specific task (i.e., the task for which the NER model is to betrained). After applying the seeding rules to the unstructured textusing the seeding rules, the weakly-labeled data is used to train aninitial iteration of an artificial neural network-based NER model. Theunstructured text is also processed to automatically identify aplurality of potential rules for labelling “named-entities.” Theautomatically identified rules are applied to the unstructured text andthe text/label combinations determined by the rules are compared to thetext/label combinations determined by the initial iteration of the NERmodel. The most successful “rules” are identified using a scoring metricand are then applied to the original unstructured text to generateanother set of training data. The NER model is then retrained based onthe data as labeled by the new set of selected rules. This trainingprocess is iteratively repeated to continue to refine and improve theNER model.

In some implementations, the “weakly supervised” mechanism for trainingthe NER model uses bootstrapping to generate weakly labeled data withsymbolic rules and also automatically trains the NER model to recognizeentities with neural representations. For example, in someimplementations, the initial seeding rules may include a rule such as“located in ______” to explicitly identify at least some locations inthe unstructured text. In addition, by comparing low-dimension neuralrepresentations (i.e., word embeddings) and iteratively retraining theNER model, the NER model can be trained to identify new entities. Theframework described in the examples below uses both explicit logicalrules and neural representations to find new entities from an unlabeledcorpus (e.g., the unstructured text) iteratively. Also, because thesystems and methods use logical rules to obtain weak labels andrecognize entities, each system prediction provided by the trained NERmodel can be traced back to original logical rules, which makes theprediction results explainable.

In one embodiment, the invention provides a method for training amachine-learning model to perform named-entity recognition. All possibleentity candidates and all possible rule candidates are automaticallyidentified in an input data set of unlabeled text. An initial trainingof the machine-learning model is performed by applying a set of seedingrules to the input data set to assign labels to the entity candidatesand using the label assignments as a first set of training data. Thetrained machine-learning model is then applied to the unlabeled text anda subset of rules from the rule candidates is identified that produceslabels that most accurately match the labels assigned by the trainedmachine-learning model. The machine-learning model is then retrainedusing the labels assigned by the identified subset of rules as thesecond set of training data. The process of applying the retrainedmodel, identifying a subset of rules that assign labels that mostaccurately match the labels assigned by the retrained model, andperforming an additional retraining of the model are iterativelyrepeated to further refine and improve the performance of themachine-learning model for named-entity recognition.

In another embodiment the invention provides a system for training amachine-learning model to perform named-entity recognition. The systemincludes an electronic processor that is configured to identify allpossible entity candidates and all possible rule candidates in an inputdata set of unlabeled text. The electronic processor performs an initialtraining of the machine-learning model by applying a set of seedingrules to the input data set to assign labels to the entity candidatesand using the label assignments as a first set of training data. Theelectronic processor then applies the trained machine-learning model tothe unlabeled text and a subset of rules from the rule candidates isidentified that produces labels that most accurately match the labelsassigned by the trained machine-learning model. The electronic processorthen retrains machine-learning model using the labels assigned by theidentified subset of rules as the second set of training data. Theprocess of applying the retrained model, identifying a subset of rulesthat assign labels that most accurately match the labels assigned by theretrained model, and performing an additional retraining of the modelare iteratively repeated to further refine and improve the performanceof the machine-learning model for named-entity recognition.

Other aspects of the invention will become apparent by consideration ofthe detailed description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system for training and using amachine-learning-based named entity recognition (NER) mechanismaccording to one embodiment.

FIG. 2 is a schematic diagram of one example of the NER mechanismtrained and/or applied using the system of FIG. 1 .

FIG. 3 is a schematic diagram of a system framework for training the NERmechanism in the system of FIG. 1 using automatically generated logicalrules.

FIG. 4 is a flowchart of a method for training the NER mechanism of FIG.2 using the system framework of FIG. 3 .

DETAILED DESCRIPTION

Before any embodiments of the invention are explained in detail, it isto be understood that the invention is not limited in its application tothe details of construction and the arrangement of components set forthin the following description or illustrated in the following drawings.The invention is capable of other embodiments and of being practiced orof being carried out in various ways.

FIG. 1 illustrates an example of a computer-based system 100 that may beconfigured for training a named-entity recognition (NER)machine-learning mechanism, for applying a trained NER mechanism, orboth. The system 100 includes an electronic processor 101 and anon-transitory computer-readable memory 103. The memory 103 stores dataand computer-executable instructions that are accessed and executed bythe electronic processor 101 to provide the functionality of the system100 including, for example, the functionality described herein below.The electronic processor 101 is communicative coupled to a display 105and a user input device 107 (e.g., a keyboard, mouse, touch screen,etc.) to provide a user interface for operating the system 100 and fordisplaying data to a user. The electronic processor 101 is alsocommunicatively coupled to an input/output device 109 (e.g., a wired orwireless communication interface) for communicating with othercomputer-based systems.

FIG. 2 illustrates an example of a machine learning model trained and/orapplied by the system 100 of FIG. 1 . The machine learning model 201 isconfigured to receive as input unstructured and unlabeled textincluding, for example, a textual sentence 203. In response to receivingthe input text, the machine learning model is configured to output eachof a plurality of “spans” of text from the sentence and a label assignedfor each span. The labels output by the machine learning model 201indicate whether the span has been identified as a named-entity and, ifso, which category label has been assigned to the span. In the exampleof FIG. 2 , the machine learning model has identified three differentspans 205, 207, 209 in the input sentence 203. The first span 205 hasbeen assigned a “person” label 211 by the machine learning model 201indicating that the text of the first span 205 has been identified ascorresponding to the name of a person. The second span 207 has beenassigned a “location” label 213 by the machine learning model 201indicating that the text of the second span 207 has been identified ascorresponding to the name of a location (e.g., a building, a city, astate, a country, etc.). Lastly, the third span 209 has been assigned a“NEG” label 215 by the machine learning model 201. As described infurther detail below, the “NEG” label 215 indicates that the machinelearning model 201 has determined that the text of the third span 209does not correspond to any named entity.

Each “span” may include a single word from the input sentence or acombination of multiple words from the input sentence. For example, ifthe sentence “I like running” were provided as input to the machinelearning model 201, the machine learning model in some implementationsmay be configured to produce the following spans as output: [I], [like],[running], [I like], [like running], and [I like running]. Although thespecific example of FIG. 2 shows the machine learning model 201producing only 3 spans as output, in other implementations, the machinelearning model 201 may be configured to output more or fewer identifiedspans. Similarly, in some implementations, the number of spans producedas output may vary depending on the sentence that is provided as input.In fact, in some implementations, the machine learning model 201 may beconfigured to produce as output all possible spans in the inputsentence.

As a further example, if the sentence “George lives in London” wereprovided as the input 203, a trained machine learning model 201 may beconfigured to produce as output the following combinations of spans andlabels:

TABLE 1 SPAN LABEL [George] PER (“person”) [lives] NEG [in] NEG [London]LOC (“location”) [George lives] NEG [George lives in] NEG [George livesin London] NEG [lives in] NEG [lives in London] NEG [in London] NEG

FIG. 3 illustrates an example of a framework for training themachine-learning model 201 of FIG. 2 to perform named-entity recognitionand labeling for an unstructured text input. An entity candidategenerator 301 and a rule candidate generator 303 are both applied to aset of unlabeled training data 305. The entity candidate generator 301is configured to automatically process the input text 305 to identifyall possible candidates (e.g., “spans”) in the input text 305.Similarly, the rule candidate generator 303 is configured toautomatically generate potential “candidate rules” from the unlabeleddata that might be used to determine whether or not a particular “span”is a named entity.

The entity candidates and rule candidates are provided as input to aniterative NER training module 307. A rule labeler 309 automaticallyapplies a set of labeling rules 311 to each entity candidate and assignslabels to the entity candidates. As described in further detail below,on the first iteration of the iterative NER training module 307, thelabeling rules 311 includes a basic set of seeding rules 313. Thelabeled data from the rule labeler 309 is then provided as the traininginput for a neural NER model 315. The original unlabeled data 305 isthen provided as input data to the trained neural NER model 315 toproduce a “predicted data” output. The predicted data includes anidentification of one or more spans and a label assigned to the span bythe trained neural NER model 315 (see, e.g., Table 1 above). A ruleselector 317 is then configured to score and select the most accuratelabelling rules from the set of rule candidates (generated by the rulecandidate generator 303) by applying the rule candidates to theunlabeled data and comparing the results of each rule to the predicteddata output by the neural NER model 315.

The set of rule candidates that have been identified by the ruleselector 317 as being the most accurate are then used as the labelingrules 311 for the next iteration. In the next iteration of the iterativeNER training module 307, the rule labeler 309 applies the selected setof rules to the entity candidates to produce a new set of labeled dataand the new set of labeled data is used as training data to retrain theneural NER model 315. The updated neural NER model 315 is then appliedto the unlabeled data 305 to produce a new set of predicted data and therule selector 317 identifies the set of rule candidates that produceresults that most accurately match the output of the updated neural NERdata 315. In various implementations, this iterative process 307 isrepeated until an exit condition is reached (e.g., after a definednumber of iterations, after a defined performance metric is achieved, oruntil the rule selector 317 converges on a particular set of labelingrules).

FIG. 4 illustrates an example of a method applied by the system of FIG.1 to train a machine-learning mechanism (e.g., an artificial neuralnetwork) using the framework of FIG. 3 to perform named-entityrecognition and labeling (e.g., as illustrated in the example of FIG. 2). The system 100 begins by applying the entity candidate generator 301to the unlabeled data 305 to identify all entity candidates (step 401)and applies the rule candidate generator 303 to the unlabeled data toidentify all rule candidates (step 403). Next, the system 100 applies aset of seeding rules 213 to the entity candidates to label the entitycandidates (step 405) and uses this “weakly labeled” data set astraining data to train a neural NER model 315 (step 407). The system 100then applies the initial trained neural NER model 315 to the originalunlabeled data 305 (step 409) to produce a set of “predicted data”labels. Each rule candidate from the rule candidate generator 303 isalso applied to the original unlabeled data (step 411) and the accuracyof each rule candidate is scored (step 413) by comparing the results ofthe rule candidate to the set of “predicted data” labels” from theinitial trained neural NER model 315. The system identifies the topperforming rule candidates (e.g., the rule candidates that producelabels that most accurately match the set of “predicted data” labelsproduced by the neural NER model) (step 415). The system 100 thenapplies this identified set of top performing rule candidates to theoriginal unlabeled data as the new labeling rules (step 417) andretrains the neural NER model using the data labeled by the new set oflabeling rules as the training data (step 407). The retrained neural NERmodel 315 is then applied to the original unlabeled data to produce anew set of “predicted data” labels (step 409) and each rule candidate isscored again—this time by comparing the labels generated by each rulecandidate to the new set of predicted data labels generated by theretrained neural NER model (step 413).

After each iteration, the system 100 determines whether a targetperformance of the neural NER model 315 has been achieved (step 419). Ifnot, then the system 100 performs another iterative retraining of theneural NER model 315. However, once the system 100 determines that thetarget performance has been achieved, the training is complete (step421). In some implementations, the neural NER model 315 can then befurther trained using the method of FIG. 4 and a different set ofunlabeled data while, in other implementations, the trained neural NERmodel 315 may be ready for use.

The training framework of FIG. 3 takes advantage of both explicitlogical rules and neural representations to find new entities from anunlabeled corpus iteratively. Because the framework uses logical rulesto obtain “weak” labels and recognizes entities, each system predictioncan be traced back to original logical rules, which makes the predictionresults explainable.

In some implementations, the rule candidate generator 303 is configuredto use rule templates (e.g., atomic rules and composed rules) in orderto extract possible rule candidates from the unlabeled data 305. “Atomicrules” are rules that can be used to depict one signal aspect of acandidate entity while “composed rules” are rules that can be used tomatch multiple aspects of an entity. In some implementations, atomicrule r_(ti) is the atomic matching logic generated from a rule templatet_(i). Every atomic rule is associated with an entity label. Examples ofatomic rule templates include: (1) SurfaceForm (surface name matchingwith a given full name of entities (e.g., if x match “London,” then x isa LOC)), (2) Prefix (matching the prefix of a candidate span (e.g., if xmatch “Lon*,” then x is a LOC)), (3) Suffix (matching the suffix of acandidate span (e.g., if x match “*don,” then x is a LOC)), (4) PreNgram(matching the left context of a candidate span (e.g., if “located in x”,then x is a LOC), (4) PostNgram0 (matching the right context of acandidate span (e.g., if “x town”, the x is a LOC)), (5) POStag(matching the part-of-speech pattern of a candidate span), and (6)PreDependency (the parent and siblings of a span on it dependency tree).

Consider, for example, the following sentence: “The new company iscalled AdOn GmbH and is located in Hamburg.” If we use a PreNgram rule“company is called {*}”, then we will match the following spans: [AdOn],[AdOn GmbH], [AdOn GmbH and], etc. until up to the maximum length of thespan. Accordingly, the use of only atomic rules would introduce many“noisy” spans (i.e., spans that are incorrectly identified as “namedentities” by the atomic rule).

Composed rules are a composition of multiple atomic rules by logicalconjunction “{circumflex over ( )}”, logical disjunction “v”, or otherlogical operators, which are formulated as:r _(composed)=

(r ₁ , r ₂ , . . . , r _(n))  (1)Where r₁, r₂, . . . , r_(n) are atomic rules and

is a logical function to connect the atomic rules. Consider again thesentence: “The new company is called AdOn GmbH and is located inHamburg.” If we have a composed rule “(company is called {*}, PROPN)”from the template (PreNgram{circumflex over ( )}POStag), where “PROPN”denotes the part-of-speech tags for proper nouns, we will exactly matchwith the entity [AdOn GmbH].

Accordingly, in some implementations, for every candidate entity, therule candidate generator 303 will extract all of its rules according tothe given rule templates. The effective rules for different domains maybe different. Therefore, the system could potentially be configured touse different types of rules for different target domains. For example,in some biomedical domain datasets, prefix and suffix rules are moreefficient rule templates than part-of-speech tags. In someimplementations, the framework illustrated in the example of FIG. 3above allows the user to customize their rule templates according totheir dataset and domain.

As discussed above, the rule labeler 309 is configured to receive a setof unlabeled candidate entities (i.e., spans) & a set of labeling rules311 and to apply the labeling rules on unlabeled spans to obtain weaklylabeled data. In some situations, it is possible that different rulesmay produce different labels for the same candidate entity. Accordingly,in some implementations, the system 100 is configured to use a majorvoter method to deal with rule conflicts. For example, if a candidateentity is matched with three rules in total and two rules label thecandidate entity as a “location” while the third rule labels the entityas an “organization,” system 100 will assign the “location” label tothis candidate entity using majority voting. In some implementations, ifan equal number of rules apply each different label to the candidateentity (e.g., a “tie”), the system 100 would be configured to label thecandidate entity as “ABSTAIN” which means that this candidate entitywould not be assigned a label for training the neural NER model 315.

As discussed above in reference to FIG. 2 , in some implementations, themachine learning model 201 (e.g., the neural NER model 315) isconfigured to produce as output a set of different spans and a labelassigned to each output span. Given a sentence x=[w₁, w₂, . . . , w_(n)]of n tokens, a span s_(i)=[w_(b) _(i) , w_(b) _(i+1) , . . . w_(e) _(i)], where b_(i) and e_(i) are the start and end indices respectively, aspan can be represented by two components: a content representationz_(i) ^(c) calculated as the weighted average across all tokenembeddings in the span, and a boundary representation z_(i) ^(u) thatconcatenates the embeddings at the start and end positions of the span.Specifically:c ₁ , c ₂ , . . . , c _(n)=TokenRepr(w ₁ , w ₂ , . . . , w _(n))  (2)u ₁ , u ₂ , . . . , u _(n)=BiLSTM(c ₁ , c ₂ , . . . , c _(n))  (3)z _(i) ^(c)=SelfAttn(c _(b) _(i) , c _(b) _(i+1) , . . . c _(e) _(i))  (4)z _(i) ^(u) =[u _(b) _(i) ;u _(e) _(i) ]  (5)z _(t) =[z _(i) ^(c) ;z _(i) ^(u)]  (6)where TokenRepr is an embedding layer (which can be non-contextualizedor contextualized), BiLSTM is a bi-directional LSTM layer, and SelfAttnis a self-attention layer.

In some implementations, the neural NER model 315 is configured topredict labels for all spans up to a fixed length of l words using amultilayer perceptron (MLP):o _(i)=softmax(MLP^(span)(z_(i)))  (7)where o_(i) is the prediction for the span. As discussed above, in someimplementations, a negative label NEG is used as an additional label toindicate invalid spans (e.g., spans that are not named entities in theunlabeled data).

As discussed above, a rule candidate generator 303 is configured togenerate all candidate rules from unlabeled data using pre-defined ruletemplates. In some implementations of the learning framework of FIG. 3 ,the system is configured to automatically select new labeling rules fromthe rule candidates using a rule selector 317. In each iteration, therule selector 317 scores all candidate rules and selects the top-scoredrules as new labeling rules 311. For example, in some implementations,the system first estimates the possible labels for all candidate spansusing the trained neural NER model 315 (i.e., the set of “predicteddata” labels), and then scores each candidate rule by comparing theresults of each rule to these weakly estimated labels. In someimplementations, a score for each candidate rule r_(i) may be calculatedusing the equation:

$\begin{matrix}{{F\left( r_{i} \right)} = {\frac{F_{i}}{N_{i}} \times \log_{2}F_{i}}} & (8)\end{matrix}$where F_(i) is the number of category members extracted by rule r_(i)(i.e., “correctly” labeled spans) and N_(i) is the total number of spansextracted by rule r_(i). This method considers both the precision andrecall of rules because the

$\frac{F_{i}}{N_{i}}$component is the precision score of the rule and the log₂F_(i) componentrepresents the rules' ability to categorize more spans. For example, ifa rule r_(i) matches 100 instances (N₁=100) and 80 of the spans thatmatch the rule were also assigned the same label by the neural NER model315 (F₁=80), then the score for the rule r₁ would be F(r₁)=5.06.

In some implementations, the system is configured to identify a definednumber (N) of the top scoring rules for each rule template and for eachentity category as the new labeling rules for the next iteration. Insome implementations, the system is configured to use N=5 for the firstiteration. In some implementations, the system is also configured toprevent low precision rules from being added to the pool of labelingrules by setting a threshold (r=0.8) for precision of rules. This methodallows a variety of patterns to be considered yet is precise enough thatall of the patterns are strongly associated with the entity category.

Accordingly, the systems and methods described in the examples aboveprovide, among other things, a mechanism for weakly supervised trainingof a machine-learning-based named-entity recognition (NER) model byiteratively scoring a set of automatically generated rule candidatesagainst the trained NER model and using the highest scoring rulecandidates to generate training data labels for a subsequent retrainingiteration of the NER model. Features and advantages of this inventionare set forth in the following claims.

What is claimed is:
 1. A method of training a machine-learning model toperform named-entity recognition of unlabeled text data, themachine-learning model being configured to receive the unlabeled textdata as input and to produce as output one or more text spans identifiedas named-entities and a category label assigned to each of the one ormore text spans identified as named-entities, the method comprising:identifying, by an electronic processor, a plurality of entitycandidates from an input data set of unlabeled text, each entitycandidate of the plurality of entity candidates including a text spanfrom the input data set; applying, by the electronic processor, a set ofseeding rules to the input data set to assign a label to each entitycandidate of the plurality of entity candidates; training themachine-learning model using the labels assigned to the plurality ofentity candidates using the set of seeding rules as a first set oftraining data; applying the trained machine-learning model to theunlabeled text of the input data set to produce a first set of predicteddata, the first set of predicted data including one or more text spansfrom the unlabeled text identified as named-entities and a categorylabel assigned to each of the one or more text spans identified asnamed-entities; automatically generating, by the electronic processor, aplurality of rule candidates from the input data set of unlabeled text,wherein each rule candidate of the plurality of rule candidates isgenerated using a set of rule templates; applying the plurality of rulecandidates to the unlabeled text of the input data set to produce a setof label candidate data; identifying a subset of rules from theplurality of rule candidates that, when applied to the unlabeled text,assign labels to entity candidates that most accurately match theassigned labels in the first set of predicted data; and retraining themachine-learning model using the labels assigned by the identifiedsubset of rules as a second set of training data.
 2. The method of claim1, further comprising: applying the retrained machine-learning model tothe unlabeled data of the input data set to produce a second set ofpredicted data, identifying a second subset of rules from the pluralityof rule candidates that, when applied to the unlabeled text, assignlabels to entity candidates that most accurately match the assignedlabels in the second set of predicted data; and further retraining themachine-learning model using the labels assigned by the identifiedsecond subset of rules as a third set of training data.
 3. The method ofclaim 1, further comprising iteratively retraining the machine-learningmodel by repeating the acts of applying the retrained machine-learningmodel to the unlabeled text of the input data set; identifying asubsequent subset of rules from the plurality of rule candidates that,when applied to the unlabeled text, assign labels to entity candidatesthat most accurately match the labels assigned by the retrainedmachine-learning model; and performing an additional iterativeretraining of the machine-learning model using the labels assigned bythe subsequent subset of rules as the subsequent set of training data.4. The method of claim 1, wherein identifying, by the electronicprocessor, the plurality of entity candidates from the input data set ofunlabeled text includes identifying as entity candidates: text spans ofeach individual word in the unlabeled text; and text spans of eachsequence of two or more sequentially adjacent words in the unlabeledtext.
 5. The method of claim 1, wherein applying the set of seedingrules to the input data set includes applying a small set of predefinedsymbolic rules to the input data set.
 6. The method of claim 1, whereinautomatically generating the plurality of rule candidates from the inputdata set of unlabeled text includes generating, for each entitycandidate of the plurality of entity candidates, every possible ruledefined by each rule template in the set of rule templates.
 7. Themethod of claim 1 wherein automatically generating the plurality of rulecandidates from the input data set of unlabeled text includes generatinga plurality of atomic rules and a plurality of combination rules foreach of a plurality of category labels, wherein each atomic rule of theplurality of atomic rules assigns the category label to each entitycandidate that meets a single matching criterion defined by the atomicrule, and wherein each combination rule of the plurality of combinationrules assigns the category label to each entity candidate that meetsmatching criteria defined by a different combination of two or moreatomic rules.
 8. The method of claim 1, wherein identifying the subsetof rules from the plurality of rule candidates that assigns labels thatmost accurately match the assigned labels in the first set of predicteddata includes: assigning a score to each rule candidate based on anumber of entity candidates that are assigned a same label by the rulecandidate and by the trained machine-learning model; and including inthe identified subset of rules a defined number of highest scoring rulesfor each category label of a plurality of category labels.
 9. The methodof claim 1, wherein identifying the subset of rules from the pluralityof rule candidates that assigns labels that most accurately match theassigned labels in the first set of predicted data includes assigning ascore to each rule candidate based at least in part on a ratio of$\frac{F_{i}}{N_{i}},$ where F_(i) is a number of entity candidates thatare assigned a same label by the rule candidate and by the trainedmachine-learning model and where N_(i) is a total number of entitycandidates that are assigned the same label by the rule candidate. 10.The method of claim 9, wherein assigning the score to each rulecandidate further includes assigning a score to each rule candidatebased on a formula:${F\left( r_{i} \right)} = {\frac{F_{i}}{N_{i}} \times \log_{2}F_{i}}$where F(r_(i)) is the score assigned to a rule candidate r_(i).
 11. Acomputer-based system for training a machine-learning model to performnamed-entity recognition of unlabeled text data, the machine-learningmodel being configured to receive the unlabeled text data as input andto produce as output one or more text spans identified as named-entitiesand a category label assigned to each of the one or more text spansidentified as named-entities, the system comprising an electronicprocessor configured to: identify a plurality of entity candidates froman input data set of unlabeled text, each entity candidate of theplurality of entity candidates including a text span from the input dataset; apply a set of seeding rules to the input data set to assign alabel to each entity candidate of the plurality of entity candidates;train the machine-learning model using the labels assigned to theplurality of entity candidates using the set of seeding rules as a firstset of training data; apply the trained machine-learning model to theunlabeled text of the input data set to produce a first set of predicteddata, the first set of predicted data including one or more text spansfrom the unlabeled text identified as named-entities and a categorylabel assigned to each of the one or more text spans identified asnamed-entities; automatically generate a plurality of rule candidatesfrom the input data set of unlabeled text, wherein each rule candidateof the plurality of rule candidates is generated using a set of ruletemplates; apply the plurality of rule candidates to the unlabeled textof the input data; identify a subset of rules from the plurality of rulecandidates that, when applied to the unlabeled text, assign labels toentity candidates that most accurately match the assigned labels in thefirst set of predicted data; and retrain the machine-learning modelusing the labels assigned by the identified subset of rules as a secondset of training data.
 12. The system of claim 11, wherein the electronicprocessor is further configured to: apply the retrained machine-learningmodel to the unlabeled data of the input data set to produce a secondset of predicted data; identify a second subset of rules from theplurality of rule candidates that, when applied to the unlabeled text,assign labels to entity candidates that most accurately match theassigned labels in the second set of predicted data; and further retrainthe machine-learning model using the labels assigned by the identifiedsecond subset of rules as a third set of training data.
 13. The systemof claim 11, wherein the electronic processor is further configured toiteratively retrain the machine-learning model by repeatedly applyingthe retrained machine-learning model to the unlabeled text of the inputdata set; identifying a subsequent subset of rules from the plurality ofrule candidates that, when applied to the unlabeled text, assign labelsto entity candidates that most accurately match the labels assigned bythe retrained machine-learning model; and performing an additionaliterative retraining of the machine-learning model using the labelsassigned by the subsequent subset of rules as the subsequent set oftraining data.
 14. The system of claim 11, wherein the electronicprocessor is configured to identify the plurality of entity candidatesfrom the input data set of unlabeled text by identifying as entitycandidates: text spans of each individual word in the unlabeled text;and text spans of each sequence of two or more sequentially adjacentwords in the unlabeled text.
 15. The system of claim 11, wherein theelectronic processor is configured to apply the set of seeding rules tothe input data set by applying a small set of predefined symbolic rulesto the input data set.
 16. The system of claim 11, wherein theelectronic processor is configured to automatically generate theplurality of rule candidates from the input data set of unlabeled textby generating, for each entity candidate of the plurality of entitycandidates, every possible rule defined by each rule template in the setof rule templates.
 17. The system of claim 11 wherein the electronicprocessor is configured to automatically generate the plurality of rulecandidates from the input data set of unlabeled text by generating aplurality of atomic rules and a plurality of combination rules for eachof a plurality of category labels, wherein each atomic rule of theplurality of atomic rules assigns the category label to each entitycandidate that meets a single matching criterion defined by the atomicrule, and wherein each combination rule of the plurality of combinationrules assigns the category label to each entity candidate that meetsmatching criteria defined by a different combination of two or moreatomic rules.
 18. The system of claim 11, wherein the electronicprocessor is configured to identify the subset of rules from theplurality of rule candidates that assigns labels that most accuratelymatch the assigned labels in the first set of predicted data by:assigning a score to each rule candidate based on a number of entitycandidates that are assigned a same label by the rule candidate and bythe trained machine-learning model; and including in the identifiedsubset of rules a defined number of highest scoring rules for eachcategory label of a plurality of category labels.
 19. The system ofclaim 11, wherein the electronic processor is configured to identify thesubset of rules from the plurality of rule candidates that assignslabels that most accurately match the assigned labels in the first setof predicted data bv assigning a score to each rule candidate based atleast in part on a ratio of $\frac{F_{i}}{N_{i}},$ where F_(i) is anumber of entity candidates that are assigned a same label by the rulecandidate and by the trained machine-learning model and where N_(i) is atotal number of entity candidates that are assigned the same label bythe rule candidate.
 20. The system of claim 19, wherein the electronicprocessor is further configured to assign the score to each rulecandidate by assigning a score to each rule candidate based on aformula:${F\left( r_{i} \right)} = {\frac{F_{i}}{N_{i}} \times \log_{2}F_{i}}$where F(r_(i)) is the score assigned to a rule candidate r_(i).